Vehicular system for testing performance of headlamp detection systems

ABSTRACT

A method for testing a vehicular driving assist system includes providing a neural network and training the neural network using a database of images, with each image of the database of images including an image of a headlight or a taillight of a vehicle. The trained neural network is provided with an input image that does not include a headlight or a taillight. The neural network, using the input image, generates an output image, with the output image including an image of a headlight or taillight generated by the neural network. The output image is provided as an input to the driving assist system to test the driving assist system.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the filing benefits of U.S. provisionalapplication Ser. No. 62/955,548, filed Dec. 31, 2019, which is herebyincorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle vision system for avehicle and, more particularly, to a vehicle vision system that utilizesone or more cameras at a vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,796,094; 5,670,935 and/or 5,550,677, which are herebyincorporated herein by reference in their entireties.

SUMMARY OF THE INVENTION

The present invention provides a method for testing a vehicular drivingassistance system or vision system or imaging system that utilizes oneor more cameras to capture image data representative of a field of viewexterior of the vehicle. The method includes providing a neural networkand training the neural network using a database of images. Each imageof the database of images includes a headlight or a taillight of avehicle. The method also includes providing to the trained neuralnetwork an input image that does not include a headlight or a taillightand generating, by the neural network, using the input image, an outputimage. The output image includes a headlight or taillight generated bythe neural network. The method also includes providing the output imageas an input to the driving assist system to test the driving assistsystem.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a vision system thatincorporates cameras in accordance with the present invention;

FIG. 2 is a schematic view of a testing system in accordance with thepresent invention; and

FIG. 3 is a schematic view of a generative adversarial network inaccordance with the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle vision system and/or driver or driving assist system and/orobject detection system and/or alert system operates to capture imagesexterior of the vehicle and may process the captured image data todisplay images and to detect objects at or near the vehicle and in thepredicted path of the vehicle, such as to assist a driver of the vehiclein maneuvering the vehicle in a rearward direction. The vision systemincludes an image processor or image processing system that is operableto receive image data from one or more cameras and provide an output toa display device for displaying images representative of the capturedimage data. Optionally, the vision system may provide a display, such asa rearview display or a top down or bird's eye or surround view displayor the like.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vision system or driving assist system 10 for a vehicle 12includes at least one exterior viewing imaging sensor or camera, such asa forward viewing imaging sensor or camera, which may be disposed at andbehind the windshield 14 of the vehicle and viewing forward through thewindshield 14 so as to capture image data representative of the sceneoccurring forward of the vehicle 12 (FIG. 1). Optionally, the system mayinclude multiple exterior viewing imaging sensors or cameras, such as aforward viewing camera at the front of the vehicle, and asideward/rearward viewing camera at respective sides of the vehicle, anda rearward viewing camera at the rear of the vehicle, which captureimages exterior of the vehicle. The camera or cameras each include alens for focusing images at or onto an imaging array or imaging plane orimager of the camera. The forward viewing camera disposed at thewindshield of the vehicle views through the windshield and forward ofthe vehicle, such as for a machine vision system (such as for trafficsign recognition, headlamp control, pedestrian detection, collisionavoidance, lane marker detection and/or the like). The vision system 10includes a control or electronic control unit (ECU) having electroniccircuitry and associated software, with the electronic circuitryincluding a data processor or image processor that is operable toprocess image data captured by the camera or cameras, whereby the ECUmay detect or determine presence of objects or the like and/or thesystem may provide displayed images at a display device for viewing bythe driver of the vehicle. The data transfer or signal communicationfrom the camera to the ECU may comprise any suitable data orcommunication link, such as a vehicle network bus or the like of theequipped vehicle.

Modern vehicles often use vehicular vision systems (e.g., camerasystems) to detect and identify or recognize objects. For example, thevehicle may identify headlights and/or tail lights of vehicles for manyadvanced driving assistance systems (ADAS) (e.g., adaptive cruisecontrol (ACC), autonomous emergency braking (AEB), automatic high beams,etc.). The vision system may provide a light type and position to thevarious ADAS modules, for example to switch from high beams to lowbeams.

To test functionality of such ADAS modules, typically a large amount ofdata is collected at various speeds, distances, weather conditions,orientations, and angles of approach of the object or headlight and/ortail light to be identified as oncoming targets, preceding targets, etc.relative to the ADAS module undergoing functionality testing.Additionally, when testing on a test track, an accurate GPS or similarsystem is used to precisely track the vehicle's position (i.e., providea precise ground truth). However, this testing incurs heavy costs,considerable time, and is limited in the number of combinations of testvariations.

Aspects of the present invention include a driving assistance testsystem that uses artificial intelligence and image processing techniquesto modify images captured by a camera to emulate the effect of variousheadlights or tail lights. For example, using machine learning, such asneural networks (e.g., Generative Adversarial Networks (GAN)), theemulated effect is realistic enough to use as a real world scenario forthe ADAS module under test. The vehicular vision system may use thesegenerated images and videos to test the ADAS algorithm.

The first part of the system is a training phase 200, where, such asshown in FIG. 2, the system trains a deep learning algorithm 204 (e.g.,a neural network such as a GAN or a Convolution Neural Networks or othersuitable Machine Learning algorithm) using a database 202 of videorecordings or images. The video recordings include images of headlightsand tail lights of different vehicles or target vehicles at differentdistances in different environmental and weather conditions,orientations, and angles of approach. For example, the database 202 mayinclude a catalog of video recordings or images of various speeds,targets, and target orientations (including relevant data as to thedistance and angle between the camera and the headlights/tail lights andrelative speeds) along with accurate GPS data. The GPS information maybe used to emulate the effect of various speeds and distances. Thealgorithm 204 outputs a trained model 206. The size of the dataset maybe significantly smaller than the dataset that would be required to testthe ADAS module via typical testing.

Optionally, the model may be trained with the same image multiple timesemulating different conditions (e.g., by modifying the GPS data or othersimulated sensor data). Optionally, the images may be perturbed ormodified to enlarge the training set size without needing to acquireadditional images. For example, images may be cropped, rotated,inverted, etc.

The second part of the system is the testing phase 208, where, thetrained model 206 is provided to a deep learning algorithm or inferenceengine 212, which is provided a video recording 210 with no headlight ortail light (test images). The trained model 206, based on the training200, modifies the test image or input image 210 to have a headlight ortail light 214. This modified image 214 is used for testing the ADASmodule (e.g., a light detection module), thus generating a wide varietyof test images for the ADAS module without the need to actually capturethe images in the wide variety of environments necessary for adequatetesting of the ADAS module.

Referring now to FIG. 3, the deep learning algorithm 212 of the testingphase or system 208 may include a deep neural network called aGenerative Adversarial Network 300 (GAN) or a similar variant. The GANcomprises two sub networks—a generator or generative network 306 and adiscriminator or discriminative network 308. The GAN 300 is initiallytrained using input images 302 (i.e., images with no headlight or taillight such as images 210) and output images 304 (i.e., a database oftrained images such as database 202). The input and output images do notneed to be in pairs for the initial training, nor do the images needpixel to pixel correspondence. The generator model (corresponding to thegenerator network 306) attempts to generate images 307 which lookvisually similar to the output of the physical imager (i.e., to generateimages that do not include headlights or tail lights based on thetrained images). The discriminator model (corresponding to thediscriminator network 308) attempts to discriminate between the inputimages 302 and the generated images 307. This is an iterative processwith the discriminator discriminating and providing feedback 310 to thegenerator and the generator producing images which look increasinglyvisually similar to that of an image from the physical sensor (i.e., tothat of an image captured by a camera that includes a headlight or ataillight) until the discriminator is unable to distinguish betweengenerated images and input images. That is, at step 312 the “No” path isfollowed for the correct generated image 314 (such as images 214) as thediscriminator network 308 is unable to discriminate or determine whichimage is the test image and which image is the generated image (i.e.,the generator has “fooled” the discriminator). At this point, trainingis complete and a stopping condition for the algorithm is generated. Thecorrect generated image 314 may be provided to a suitable ADAS module(e.g., an automatic headlight control system) for testing. The ADASmodule may, for example, classify the image 314 as a headlight ortaillight and determine a position of the headlight or the taillight.

Thus, the system decreases training cost and provides unlimited testvariations by modifying an image to add a headlight or taillight throughthe use of artificial intelligence (e.g., a neural network). The systemis trained on a database of images that include a headlight or ataillight. Optionally, the training data includes GPS data to emulatethe effect of various speeds and distances. The trained system maygenerate a modified image that includes a headlight or a taillight thatmay be used to train a corresponding ADAS module (such as an automaticheadlight control system). The system allows the same video images to beused to test multiple different ADAS modules as the same or differentneural networks can make different modifications to the same videoimages depending on the ADAS module under test.

The system includes an image processor operable to process image datacaptured by the camera or cameras, such as for detecting objects orother vehicles or pedestrians or the like in the field of view of one ormore of the cameras. For example, the image processor may comprise animage processing chip selected from the EYEQ family of image processingchips available from Mobileye Vision Technologies Ltd. of Jerusalem,Israel, and may include object detection software (such as the typesdescribed in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, whichare hereby incorporated herein by reference in their entireties), andmay analyze image data to detect vehicles and/or other objects.Responsive to such image processing, and when an object or other vehicleis detected, the system may generate an alert to the driver of thevehicle and/or may generate an overlay at the displayed image tohighlight or enhance display of the detected object or vehicle, in orderto enhance the driver's awareness of the detected object or vehicle orhazardous condition during a driving maneuver of the equipped vehicle.

The vehicle may include any type of sensor or sensors, such as imagingsensors or radar sensors or lidar sensors or ultrasonic sensors or thelike. The imaging sensor or camera may capture image data for imageprocessing and may comprise any suitable camera or sensing device, suchas, for example, a two dimensional array of a plurality of photosensorelements arranged in at least 640 columns and 480 rows (at least a640×480 imaging array, such as a megapixel imaging array or the like),with a respective lens focusing images onto respective portions of thearray. The photosensor array may comprise a plurality of photosensorelements arranged in a photosensor array having rows and columns.Preferably, the imaging array has at least 300,000 photosensor elementsor pixels, more preferably at least 500,000 photosensor elements orpixels and more preferably at least 1 million photosensor elements orpixels. The imaging array may capture color image data, such as viaspectral filtering at the array, such as via an RGB (red, green andblue) filter or via a red/red complement filter or such as via an RCC(red, clear, clear) filter or the like. The logic and control circuit ofthe imaging sensor may function in any known manner, and the imageprocessing and algorithmic processing may comprise any suitable meansfor processing the images and/or image data.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 9,233,641;9,146,898; 9,174,574; 9,090,234; 9,077,098; 8,818,042; 8,886,401;9,077,962; 9,068,390; 9,140,789; 9,092,986; 9,205,776; 8,917,169;8,694,224; 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935;6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229;7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287;5,929,786 and/or 5,786,772, and/or U.S. Publication Nos.US-2014-0340510; US-2014-0313339; US-2014-0347486; US-2014-0320658;US-2014-0336876; US-2014-0307095; US-2014-0327774; US-2014-0327772;US-2014-0320636; US-2014-0293057; US-2014-0309884; US-2014-0226012;US-2014-0293042; US-2014-0218535; US-2014-0218535; US-2014-0247354;US-2014-0247355; US-2014-0247352; US-2014-0232869; US-2014-0211009;US-2014-0160276; US-2014-0168437; US-2014-0168415; US-2014-0160291;US-2014-0152825; US-2014-0139676; US-2014-0138140; US-2014-0104426;US-2014-0098229; US-2014-0085472; US-2014-0067206; US-2014-0049646;US-2014-0052340; US-2014-0025240; US-2014-0028852; US-2014-005907;US-2013-0314503; US-2013-0298866; US-2013-0222593; US-2013-0300869;US-2013-0278769; US-2013-0258077; US-2013-0258077; US-2013-0242099;US-2013-0215271; US-2013-0141578 and/or US-2013-0002873, which are allhereby incorporated herein by reference in their entireties. The systemmay communicate with other communication systems via any suitable means,such as by utilizing aspects of the systems described in U.S. Pat. Nos.10,071,687; 9,900,490; 9,126,525 and/or 9,036,026, which are herebyincorporated herein by reference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

1. A method for testing a vehicular driving assist system, the methodcomprising: providing a neural network; training the neural networkusing a database of images, wherein each image of the database comprisesan image of a headlight of a vehicle or an image of a taillight of avehicle; providing to the trained neural network an input image thatdoes not include a headlight or a taillight; generating, by the neuralnetwork using the input image, an output image, wherein the output imagecomprises an image of a headlight generated by the neural network or animage of a taillight generated by the neural network; and testing thevehicular driving assist system using the generated output image as aninput to the vehicular driving assist system.
 2. The method of claim 1,wherein the neural network comprises a generative adversarial network.3. The method of claim 2, wherein training the neural network comprisesiterating the training until a discriminator cannot distinguish betweenthe input image and the generated output image.
 4. The method of claim1, wherein the vehicular driving assist system comprises an automaticheadlight control system.
 5. The method of claim 1, wherein the databasecomprises a plurality of video recordings.
 6. The method of claim 5,wherein the plurality of video recordings comprises video recordings ofa headlight or a taillight at least one selected from the groupconsisting of (i) different speeds relative to a camera capturing thevideo recordings, (ii) different distances relative to the cameracapturing the video recordings and (iii) different orientations relativeto the camera capturing the video recordings.
 7. The method of claim 1,wherein the vehicular driving assist system classifies the generatedoutput image as including a headlight or a taillight and determines aposition of the headlight or the taillight within the output image. 8.The method of claim 1, wherein the database comprises GPS datacorresponding to the images.
 9. The method of claim 8, wherein the GPSdata emulates an effect of speed and distance experienced by a cameracapturing the images.
 10. The method of claim 1, wherein the neuralnetwork generates the output image by modifying the input image with theimage of the headlight or the taillight.
 11. A method for testing avehicular driving assist system, the method comprising: providing agenerative adversarial network (GAN); training the GAN using a databaseof video recordings, wherein each video recording of the databasecomprises a plurality of images that include a headlight of a vehicle ora taillight of a vehicle; providing to the trained GAN an input imagethat does not include a headlight or a taillight; generating, by the GANusing the input image, an output image, wherein the output imagecomprises an image of a headlight generated by the GAN or an image of ataillight generated by the GAN; and testing the vehicular driving assistsystem using the generated output image as an input to the vehiculardriving assist system.
 12. The method of claim 11, wherein training theGAN comprises iterating the training until a discriminator cannotdistinguish between the input image and the generated output image. 13.The method of claim 11, wherein the vehicular driving assist systemcomprises an automatic headlight control system.
 14. The method of claim11, wherein the database of video recordings comprises video recordingsof a headlight or a taillight at least one selected from the groupconsisting of (i) different speeds relative to a camera capturing thevideo recordings, (ii) different distances relative to the cameracapturing the video recordings and (iii) different orientations relativeto the camera capturing the video recordings.
 15. The method of claim11, wherein the vehicular driving assist system classifies the outputimage as including a headlight or a taillight and determines a positionof the headlight or the taillight within the output image.
 16. Themethod of claim 11, wherein the database comprises GPS datacorresponding to the video recordings.
 17. A method for testing anautomatic headlight control system, the method comprising: providing aneural network; training the neural network using a database of images,wherein each image of the database comprises an image of a headlight ofa vehicle or an image of a taillight of a vehicle; providing to thetrained neural network an input image that does not include a headlightor a taillight; generating, by the neural network using the input image,an output image, wherein the output image comprises an image of aheadlight generated by the neural network or an image of a taillightgenerated by the neural network; testing the vehicular driving assistsystem using the generated output image as an input to the automaticheadlight control system; and wherein the automatic headlight controlsystem classifies the generated output image as including a headlight ora taillight and determines a position of the headlight or the taillightwithin the generated output image.
 18. The method of claim 17, whereinthe neural network comprises a generative adversarial network.
 19. Themethod of claim 18, wherein training the neural network comprisesiterating the training until a discriminator cannot distinguish betweenthe input image and the generated output image.
 20. The method of claim17, wherein the database comprises a plurality of video recordings.